Pepper blossoms are sensitive to temperature fluctuations during pollination. Normal pollination and fruit set don't occur when night temperatures fall below 58°F and daytime temperatures rise above 85°F. Under these temperature conditions, the blossoms fall off, often before pollination. If pollination has occurred and the fruit has begun to set, but isn't completely fertilized at the time the blossoms drop, rough and misshapen fruit result.
Blossom drop only causes a delay in fruit production. When the temperatures are less extreme, a full crop of fruit will set, and the plants will be productive the rest of the season. Discard rough or misshapen fruit; they will never develop fully. Irrigation for cooling during hot periods can help to reduce losses.